Skip to content

Treat cluster-info read failures at worker startup as transient - #359

Merged
rmcdaniel merged 2 commits into
mainfrom
general/cluster-info-transient
Nov 20, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
general/cluster-info-transient

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

A cluster-info read failure at worker startup stopped the Python worker. Startup now treats cluster-info reads as transient.

Change

Startup discovery. GET /api/cluster/info is side-effect free, so _register_through_transients (Worker.run() and run_until()) now treats as transient:

  • any refused, reset or timed-out connection
  • any 5xx or 429 answer to discovery, unless the Server says retryable: false

Before, only the narrower set from #355 counted, so a plain 500 {"message":"Server Error"}, a 500 with a reason, or a 501 stopped the worker. Retries use the shared backoff uniform(0, min(4, 0.25·2^(n−1))) seconds, and the worker waits until discovery answers.

Escalation. This is the same as for poll outages:

  • warnings during the first 60 s
  • then an ERROR (worker registration has failed for N s: ...)
  • then an ERROR every 5 minutes while the outage lasts

Each startup is a fresh streak.

Typed refusals are unchanged. 401 and 403 on discovery stay fatal at startup, as decided, with the error-level stop log. retryable: false is fatal too.

Run time. After registration the worker uses the discovery it registered with; the cached _runtime_discovery is not refetched. The one run-time read, #358's capability recovery after an unknown digest, already keeps the answer and drops the field if discovery fails. A cluster-info outage at run time therefore cannot stop the worker.

Tests (tests/test_worker_transient_runtime.py)

  • At startup, with each of 500 {"message"}, 500 {"reason"}, 501, 429 and a refused connection on cluster info, repeated six times and then recovering:
    • the worker waits, registers and polls
    • discovery is read exactly 7 times, with no read after registration
  • Escalation: with a fake clock at 31 s per attempt, one warning comes first, then ERRORs at 62 s and 372 s.
  • Still fatal on discovery: 401, 403 and 500 retryable: false, each served once and never retried.

Local results (Python 3.14):

  • pytest tests/ -m "not integration": 7899 passed, 3 skipped.
  • ruff: clean.
  • mypy: clean.
  • regression-corpus guard: clean.

🤖 Generated with Claude Code

rmcdaniel and others added 2 commits November 20, 2026 10:11
GET /api/cluster/info has no side effects, so any connection failure
and any 5xx or 429 is now transient for the startup discovery read
(unless retryable: false); before, only the narrower transient set
was, so a plain 500 stopped the worker. Startup retries use the shared
backoff and now escalate like poll outages: ERROR after 60 s, then every
5 minutes. Typed refusals (401/403) stay fatal. After registration the
worker uses the discovery it registered with and does not read cluster
info again.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added bug Something isn't working agent:general labels Nov 20, 2026
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 2f5b951 into main Nov 20, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant